1. Field of the Invention
The present invention relates to an image processing device, an image processing method and a computer readable storage medium storing an image processing program for performing noise reduction process on image data.
2. Description of the Related Art
An imaging device is known which focuses light collected by an image capturing lens to form an image on an image sensor such as a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), and the like, and captures an image of a subject. The image captured by such an imaging system includes noise added to the image when the image is an analog signal, such as when photoelectrically converting the image formed on the image sensor through the image capturing lens at light receiving units of each pixel of the image sensor or when amplifying and A/D converting a charge amount obtained by photoelectrical conversion.
For example, the noise predominantly generated in photoelectrical conversion is shot noise which is caused by stochastic fluctuations of photoelectrical conversion, and the noise is a so-called random noise. Fluctuations of generated charge amount increase as the number of photons (or dark current) increases. On the other hand, the noise predominantly generated in A/D conversion is a so-called fixed pattern noise, which is different from the random noise, including noise added by dark current and sensitivity variations of each pixel, crosstalk of signal transmission path and peripheral pixels, and the like, and noise added by variations of signal amplification due to multi-line reading.
The fixed pattern noise can be reduced by storing characteristics of the fixed pattern noise as correction data in advance and correcting the image data after A/D conversion by using the correction data. Meanwhile, regarding the random noise, characteristics thereof cannot be estimated in advance, unlike the fixed pattern noise. Therefore, conventionally, a signal to noise ratio (SN ratio) is improved by using smoothing process in order to reduce the random noise.
Here, ds the smoothing process, there are proposed various methods from a method in which fixed low pass filter characteristics are uniformly applied regardless of local structures of captured image, to a method in which a low pass filter characteristic applied is changed to perform filter processing while considering local structures of an image, and noise reduction performance is determined depending on which method is used.
Among these methods, as a method which can effectively reduce noise, a method is known in which a weight coefficient is determined for each pixel and each pixel is smoothed by using the determined weight coefficient (for example, refer to Japanese Laid-open Patent Publication Nos. 2006-302023 and 2008-124764). In Japanese Laid-open Patent Publication No. 2006-302023, filter processing is performed while adaptively changing the weight coefficient for each pixel on the basis of correlation (for example, absolute value of pixel value difference) between pixel values of a pixel of interest and a pixel around the pixel of interest. On the other hand, in Japanese Laid-open Patent Publication No. 2008-124764, the weight coefficient is determined by using a relative spatial distance and an absolute value of pixel value difference between a pixel of interest and a pixel around the pixel of interest, and the filter processing is performed by using the determined weight coefficient. Then, a noise amount with respect to the pixel of interest is estimated on the basis of the filter-processed pixel of interest value, and the noise reduction process of the pixel of interest is performed on the basis of the pixel of interest value and the estimated noise amount.
A typical smoothing filter which uses such an absolute value of pixel value difference is a bilateral filter. The bilateral filter is an adaptive filter which uses two types of parameters and changes low pass filter characteristics in accordance with local structures of an image, and can perform smoothing process while preserving edges in the image. In this bilateral filter, a Gauss function, for example, is used as a function to determine the filter coefficient, and the filter coefficient is calculated as a product of Gauss function values which are obtained by using the two types of parameters as weights.
Here, one of the two parameters used in the bilateral filter corresponds to the absolute value of pixel value difference between the pixel of interest and a pixel around the pixel of interest, and the smaller the value is, the larger the filter coefficient is set. On the other hand, for the other parameter, a relative spatial distance between the pixel of interest and a pixel around the pixel interest is used, and the smaller the value of the relative spatial distance is, the larger the filter coefficient is set.
The parameters will be briefly described. One parameter which is the absolute value of pixel value difference is used to determine whether there is an edge or not, and the determination condition is determined by a σ value of the Gauss function. When an edge amount (specifically, absolute value of pixel value difference) desired to be preserved is sufficiently larger than the σ value, the Gauss function value approaches zero, and thus the pixel does not contribute to the smoothing process. Therefore, a pixel having a large absolute value of pixel value difference between the pixel and pixels around the pixel (a pixel which does not correlate with the absolute value of the pixel value difference), such as a pixel at an edge boundary is not used for the smoothing process. Based on this, an effect that an edge is not smoothed is obtained.
On the other hand, the other parameter which is the spatial distance is based on an assumption that a correlation between pixel values decreases as the spatial distance between the pixels increases when statistically viewing local structures of an image. In other words, regarding a second pixel having the same pixel value as that of a first pixel and being located far from the first pixel, it is assumed that the original pixel value of the second pixel is not the same as the pixel value of the first pixel but the second pixel has the same pixel value because noise is added to the pixel value. Specifically, when pixels having the sane pixel value as that of the first pixel are located near the first pixel and far from the first pixel respectively, it is assumed that the near pixel has a correlation higher than that of the far pixel, and a larger weight is applied to the near pixel. Therefore, a pixel having a large spatial distance from pixels around the pixel (pixel having no correlation in spatial distance) is not used for the smoothing process.
The noise reduction effect using the bilateral filter modeled a described above shows significantly better performance in an area formed by a sharp and large edge and a flat portion.
Other than the above, as another method of smoothing process having a high noise reduction effect, for example, there is a method in which a one-dimensional low pass filter is applied along an edge direction (for example, refer to Japanese Laid-open Patent Publication No. 61-206376) In Japanese Laid-open Patent Publication No. 61-206376, first, the direction of the edge is determined. Then, on the basis of the determination result, a smoothing filter in the horizontal direction, smoothing filter in the vertical direction, or a smoothing filter in the diagonal direction is selected to perform the filter processing.
Further, as a method for improving the noise reduction effect in the filter processing along the edge direction, a method in which a two-dimensional filter is applied along the edge direction is known (for example, refer to U.S. Pat. No. 7,317,842). In U.S. Pat. No. 7,317,842, when determining the edge direction, distribution values in a plurality of directions are calculated in a local area, and an anisotropic two-dimensional filter is generated on the basis of the determined edge direction and its distribution value.
Further, there is a method that improves determination accuracy of the edge direction and is able to preserve fine edges (for example, refer to U.S. Patent Application Publication No. 2008/107352). Here, the fine edges are edge boundary portions having a small pixel value difference (brightness value difference), and hereinafter, the fine edges are arbitrarily referred to as “microstructure”. U.S. Patent Application Publication No. 2008/107352 discloses smoothing process for an image in which spatial frequency is subbanded. Specifically, the smoothing process is performed by switching between a bilateral filter and a direction-dependent Gauss filter on the oasis of an energy amount of a local area and a determination threshold value of the energy amount. When using the direction-dependent Gauss filter, a filter coefficient is selected from a look-up table (LUT) prepared in advance, on the basis of an amount indicating isotropy of the local area, and the filter coefficient weighted along an edge direction is applied to the local area. More specifically, the bilateral filter and the direction-dependent Gauss filter are switched depending on the energy amount of the local area, that is, depending on whether there is a large edge or not, so that a control considering the characteristics of the bilateral filter is realized.